Toward self-healing distributed services

Modern clouds and datacenters are rapidly growing in scale and complexity. They contain an eclectic set of hardware, networks, applications, and virtual machines that give rise to complex emergent behaviour, unobservable when components are tested individually. Problem diagnosis in these environments is especially difficult, as it can require developers to have expert knowledge about every detail of the datacenter. For example, Google engineers diagnosing a distributed service’s performance must often have intricate knowledge of the service, its dependencies (e.g., GFS, BigTable, the authentication mechanism, and the network topology), its configuration (e.g., the machines on which it’s running and its resource allocation), its critical paths, and the many co-located applications that might be interfering with it. To keep problems in these environments from becoming completely undiagnosable, many researchers believe the only recourse is to build self-healing systems, capable of fixing problems automatically without human intervention.